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Abstract 

A subset S" of a group G is called an Engel set if, for all x,y ^ S, there 
is a non-negative integer n — n(x,y) such that [x, „?/] = 1. In this 
paper we are interested in finding conditions for a group generated by 
a finite Engel set to be nilpotent. In particular, we focus our investi- 
gation on groups generated by an Engel set of size two. 
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1 Introduction 

A subset S" of a group G is called an Engel set if, for all x, y G 5, there is a 
non-negative integer n = n{x,y) such that [x, nV] = 1- It is known that, for 
a group G satisfying Max-a6, a normal subset 5 C G is an Engel set if and 
only if it is contained in the Fitting subgroup of G (see [7], Theorem 7.23; see 
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also [T]) and so, in this case, {S) is nilpotent whenever S is finite. However, 
a group generated by a finite Engel set is not necessarily nilpotent: Golod's 
examples show that there exist infinite non-nilpotent groups generated by 
an Engel set with three or more elements (see [5]). Furthermore, if S is an 
Engel set of size three, then an easier example of a non-nilpotent group 
generated by S is the wreath product of the alternating group of degree 5 
with the cyclic group of order 3: it has a presentation of type (r, s, t) (see 
[3]), i.e. S = {a, 6, c} where (a, b) is nilpotent of class r, (a, c) is nilpotent of 
class s and (6, c) is nilpotent of class t. All these groups are not soluble, but 
the nilpotency does not hold even in the soluble case. In |3j it was shown 
that every group with a presentation of type (1,2,2) is soluble of length at 
most 3 and that there are non-nilpotent groups of this type. 

In this paper, we first get that any nilpotent-by-abelian group generated 
by a finite Engel set is nilpotent and then we focus on groups generated 
by an Engel set of size two. In particular, we prove that such a group is 
nilpotent whenever it is abelian- by- (nilpotent of class 2). This is the best 
possible result in the soluble case. In fact, we construct by GAP (see [4J) a 
non-nilpotent counterexample which is abelian- by- (nilpotent of class 3). On 
the other hand, some of the counterexamples in [3], mentioned above, are 
abelian- by- (nilpotent of class 2) and generated by an Engel set of size three. 

2 Groups that are Nilpotent-by- Abelian 

We start with a result that is certainly already known. It generalizes, for 
metabelian groups, two basic properties of commutators. 

Lemma 2.1. Let G he a metabelian group and x,y,z be elements of G. For 
all positive integers n, we have: 

(i) „y] = [x, \- 

(a) [xy, nz] = [x, nz\[x, „z,y][y, nz\- 

Proof. Since G is metabelian, every g va. G induces on G' an endomorphism 
—\ + g that maps u to u~^u^, and any two of these commute. We thus have: 



[x-\ „y] = {[x,y]-'-')^-'-'y^"-' = = [x, .y]— ^ . 



As a consequence of Lemma l2. 11 we get: 

Lemma 2.2. If G is a metabelian group generated by an Engel set S, then 
any x (z S is a left Engel element. In particular, G is locally nilpotent. 




□ 
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Proof. Take a finite subset of 5, say T = {xi, . . . ,Xr}, and suppose [xj, nXj] = 
1 for all 1 < i,j < r. By the previous lemma, every Xj is a left n-Engel 
element in G. Then (—1 + Xi)"' = 0. It follows that any product in the endo- 
morphisms —1 + Xi of weight (n — l)r + 1 is trivial. Hence (T) is nilpotent 
of class at most (n — l)r + 2. This proves that G is locally nilpotent. □ 

For a finite Engel set, we then obtain the following: 

Theorem 2.3. Let G be a nilpotent-by-abelian group generated by a finite 
Engel set. Then G is nilpotent. 

Proof. If is a normal nilpotent subgroup of G such that G/N is abelian, 
then G/N' is nilpotent by Lemma [2 .21 and so G is nilpotent by a well-known 
result of P. Hall. □ 



3 Engel sets of size two 

Let G = {x, y) be a group and assume that {x, y} is an Engel set. Then 
[x, nV] = 1 and [y, mx] = 1 for some positive integers n, m. We also say that 
the elements x and y are mutually Engel and, whenever n > m, that they 
are mutually n-Engel. If n = m = 2, then G is obviously nilpotent of class 
at most 2 and the nilpotency still holds for n = 2 and m = 3. 

Proposition 3.1. Let G = (x, y) be an arbitrary group such that [x, y,y] = 1 
and [y, x,x,x] = 1. Then G is nilpotent of class at most 3. 

Proof. By the Hall- Witt identity we have 

[[y,x],x'~\yr[x,y'\[y,x]]y[y,[y,x]-\x]^y'-^ = l, 
from which it follows 

[y,x,x"\y] = 1 

since [x, y'^ = [x, y]"^ and [y, [y, x]"^] = [x, y, y]"i = 1. Then [y, x, x, y] = 1 
and hence [y,x,x] € Z{G). Now [x,y,y] = [y,x,x] = 1 modulo Z{G), so 
G/Z{G) is nilpotent of class < 2 and G is nilpotent of class < 3. □ 

However, as we will see in the next section, this is not true in general, 
even in the soluble case. We are therefore led to consider extra conditions 
for a group generated by an Engel set of size two to be nilpotent. In the 
sequel, we will turn our attention to groups which are abelian-by- (nilpotent 
of class 2). 

Let G be any abelian-by- (nilpotent of class 2) group generated by two 
mutually Engel elements x and y. By assumption [x, „y] = 1 and [y, „x] = 1 
for some n. Suppose, by way of contradiction, that G is not nilpotent. Then 
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G has a non-nilpotent finite image by Theorem 10.51 of [7] and so we may 
assume that G is finite. 

Using induction on the order of the group, we may assume that G is 
a minimal counterexample. It follows that G contains a unique minimal 
normal subgroup A such that G/A is nilpotent. As G is not nilpotent there 
is a maximal subgroup H that is not normal. On the other hand G/A is 
nilpotent, therefore A ^ H (otherwise H/A < G/A implies that H < G). 
Thus G = AH. The group A f] H is normal in G and A D H < A. The 
minimality of A then forces AriH= 1. 

Clearly, A is an elementary abelian p-group for some prime p and H is 
nilpotent. Let P be the Sylow p-subgroup of H. Then AP/A <\ G/A and 
so AP is the Sylow p-subgroup of G. Since AP is nilpotent, we have that 
[A, AP] < A and by the minimality of A, the normal subgroup [A, AP] must 
be trivial. Thus [A, P] = I and P"^ = P^^ = p^ = p^ that is P < G. But 
A ^ P, hence P = 1 and H is a Hall p'-subgroup of G. 

Lemma 3.2. Every nontrivial element of Z{H) acts fixed point freely on A 
by conjugation. 

Proof For ah z € Z{H) and h £ H, Ga{z)'' = Ga{z) and thus Ga{z) < G. 
As (z) cannot be normal in G, we get Ga{z) = 1 by minimality of A. □ 

The next lemma shows that H is nilpotent of class 2 and that we can 
restrict our attention to n = 3. 

Lemma 3.3. Let G = AH = {x, y) he a minimal counterexample that 
is abelian-by- (nilpotent of class 2). Then A = 73(G), [x, y, y, y] = 1 and 
[y,x,x,x] = 1. 

Proof. Of course, A C 73(G) by minimality of A. Let q ^ p he a. prime. 
Then any g-subgroup of 73(G) is necessarily trivial. But G/A is a p'-group, 
therefore A = 73(G) and H is nilpotent of class 2. 

Assuming now [x, n-iu] 7^ Ij we will prove that n < 3. Let y = ah where 
a £ A,h £ H, and suppose n > 3. We have [x, y,y] £ A and n — 2 > 2, so 
that [x, n-2y] and [x, n-2y> y] lie in A. It follows that 



Notice that = aih^ with ai £ A and h = h^^ for some integer a. Thus 



[x, n~2y,y^] = [x, n~2y,yf = i- 
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[x, n-2y,y^] = [x, n-2y,aihP] = [x, n-2y,hP] 



and 



1 



[X, n-2yX''] 



[X, n-2y,h]. 



But then 



1 



[x, n-2y,ah] 



[x, n-2y,y] 



that is a contradiction. 



□ 
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We need one more preliminary lemma before proving our main result. 

Lemma 3.4. Let x = ah,y = bk where a,b & A and h,k & H. If [x,y] = 
[h, k], then 

[a, k-'^] = [b, h-^], [a, h] = 1 and [b, k] = 1, 
with a ^ 1 and b ^ I. 
Proof. We have 

[h,k] = [x,y] = [ah,bk] = [a,k]''[h,k][h,b]''. 

This implies [a,k]^'[h,b]^ = 1 and then [o, /c]'^ ^ = [6, /i]^ ^, or equiva- 
lently [a,k-^] = [b,h-'^]. 

As G ^ H we must have that one of a,b is nontrivial. Without loss of 
generality, we may assume a / 1. Clearly, [y,x,x] € A and 1 ^ [y,x] £ 
Z{H). Then 1 = [y,x.,x,x\ = [y,x,x,h] and 

= [a^t^'^'U] = [[y,x,x]'^x,h] = [x,h]. 

Thus 1 = [x,h,[y,x\] = [[a, /i]^, [y, x]] = [a,h,[y,x\]^, so [a,h] is fixed by 
[y,x]. By Lemma 13.21 it follows that [a,/i] = 1. As a consequence 5^1, 
otherwise [a,k] = 1 and [a, [/i, /c]] = 1. Arguing as for a, we then conclude 
that [b,k] = l. □ 

Theorem 3.5. Let G be any abelian-by- (nilpotent of class 2) group gener- 
ated by two mutually Engel elements x and y. Then G is nilpotent. 

Proof. Put X = a/i, y = bk where a,b € A and h,k £ H. Then [x, y] = [h, k]c 
with [h, k] E Z{H) and for some c £ A. By Lemma 13.31 we know that 

[x,y,y],[y^^^^] ^ ^ and [x,y,y,y] = [y,x,x,x] = l. 

This gives 

[x,y,y^] = l and [x,y,xP] = l. 

If {x^, yP)nA 7^ 1, the commutator [x, y] commutes with a nontrivial element 
of A. Thus [h,k] = 1 by Lemma E21 and [x,y] G A. Indeed G' < A and G 
is nilpotent by Lemma 12.21 Therefore A n {x^,y^) = 1 and we may assume 
H = {xP,yP), since {h,k) ~ {h,k)A/A = {xP,yP)A/A ~ {xP,yP). It follows 
that c must be trivial. Then 1 ^ [x,y] = [h,k] and, by Lemma 13.41 we have 

[a,k~^] = [b,h~^] and [a,h] = l, 

with a 7^ 1. 

Now, the Hall- Witt identity 

[a,k-\h]^[k,h-\a]^[h,a-\k]'' = 1 
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implies 

But [k, h~^, a] commutes with h, so [[a, k^^], h] = [[b, h~^], h] commutes with 
. Then [b,h,h]'^ = [b,h~^ ,h]~^ commutes with h'^ , in particular 
[b, h, h] commutes with h^'^'^ = Hence [b, h, h] G CA{h^'^). 

Let B = CA{h'''') and K = {h, h'''')A. Then B <] K because [h'^, k] e 
Z{H). If q is the order of h, we also have B = [b, h'^]B = [b, h]'^B. However, 
the order of [b, h] is coprime with g, thus [b, h] ^ B and [a, k~^] = [b, h~^] G 
B. So [a,k~^,h^ ] = 1 and [k,a,h] = 1. Finally, from 

[a,k,hf[k-\h^\a]''[h,a-\k-^]'' = 1, 

it follows [k, h,a] = 1 which contradicts Lemma |3.2[ □ 

When X and y are mutually 3-Engel elements, we get thanks to GAP 
that the group G in Theorem 13.51 is nilpotent of class at most 8. In fact, 
using the ANU Nilpotent Quotient package of W. Nickel (see [6]), we 
can construct the largest nilpotent quotient of G which is isomorphic to G. 

Also notice that the theorem above can be extended to a group gener- 
ated by more than two mutually Engel elements, provided that none of the 
generators has order divisible by 2 or 3. 

Corollary 3.6. Let S be a finite Engel set and assume that G = (S) is 
abelian-by- (nilpotent of class 2). If every element in S has order that is not 
divisible by 2 or 3, then G is nilpotent. 

Proof. For all x,y G 5, the subgroup (x, y) is nilpotent by Theorem 13.51 
Thus the claim follows by Proposition 1 of |3]. □ 

Using Theorem 13.51 we now present a criterion for nilpotency of a finite 
soluble group depending on information on its Sylow subgroups. 

Corollary 3.7. Let G = {x,y) be a finite soluble group with x and y mutu- 
ally Engel elements. If all Sylow subgroups of G are nilpotent of class < 2, 
then G is nilpotent. 

Proof. Let G be a counterexample of least possible order and let be 
a minimal normal subgroup of G. Then G/N is nilpotent by minimality. 
Moreover, all Sylow subgroups of G/N are nilpotent of class < 2, so that 
G/N is nilpotent of class < 2. On the other hand A^ is abelian, because G is 
soluble. Hence G is abelian-by- (nilpotent of class 2) and thus nilpotent by 
Theorem 13. 5t a contradiction. □ 



6 



4 Examples 



Our first example shows that, for any positive integer n, there exists a group 
generated by two mutually n-Engel elements which are not (n — 1)-Engel. 
This is the dihedral group of order 2^~^^. 

Example 4.1. Let consider G = {x,y \ x"^ = y"^ = 1, (xy)^" = 1). If z = xy, 
then [x,y] = and = z^ = z"^ . For any > 1, we get by induction 
[x, kV\ = z'^^'^^ and [y, fcx] = z^"^) . Therefore [x, „_iy],[y, „_ix] / 1 
whereas [x, ^y] = [y, „x] = 1. Thus x and y are mutually n-Engel elements. 
Furthermore, we have G = (y, z) and [y, = [2^, ny] = 1, so even y and z 
are mutually n-Engel elements. □ 

The following is an example obtained by GAP of a non-nilpotent group 
G generated by two mutually 3-Engel elements, for which 74(G) is abelian. 

Example 4.2. Let W = S^wrZ^ be the wreath product of the symmetric 
group of degree 3 with the cyclic group of order 4. Thus, \W\ = 2^3^. We 
have W = Q tK N, where is an elementary abelian group of order 3^ and 
Q ~ Z2 wr Z4. Moreover, Q is nilpotent of class 4. With the notation of GAP, 
let ele:=Elements(l^), x := ele[4] and y := ele[228]. Then o(x) = o(y) = 4 
and [x, 3y] = [y, 3X] = 1. As o{xy~^) = 6, the subgroup G = (x,y) of W is 
not nilpotent. Finally, one can check that G = S t< N where 5 is a group of 
order 2^ which is nilpotent of class 3. 

For completeness reasons, we point out that W = (x,y') with y' := 
ele[509] of order 6 and [x, sy'] = [y', 4X] = 1. Hence, Vl^ is a generated by 
two mutually 4-Engel elements and is not nilpotent. □ 

Notice that some more non-nilpotent groups generated by two mutually 
n-Engel elements can be found in the literature. For instance. Corollary 0.2 
of [2] says that, for n > 26, the group G{n) = (x, y | [x, „y] = [y, „x] = 1) 
is not nilpotent. We can improve upon this. In fact, we show below that 
G(4) is not soluble, because it has a quotient isomorphic to the symmetric 
group Ss- 

Example 4.3. Let 5*8 be the symmetric group of degree 8, and let x = 
(1,2,3,4)(5,6)(7,8) and y = (1, 3)(2, 5)(4, 7, 6, 8). Put x„ = [x, „y] and 
Vn = [y, nx], for any n > (so xq = x, yo = y). We then have: 

XI = (1, 6)(2, 7)(3, 8)(4, 5) yi = (1, 6)(2, 7)(3, 8)(4, 5) 

X2 = (l,5)(4,6) y2 = (2,4)(5,7) 

X3 = (1, 5)(2, 3)(4, 6)(7, 8) ys = (1, 3)(2, 4)(5, 7)(6, 8) 

X4 = (1) y4 = (1) • 

In particular, [x, 4y] = [y, 4X] = 1. However x and y are of order 4, but 
xy = (1, 5, 8, 6, 2)(3, 7, 4) is of order 15. The subgroup G = (x,y) is thus 
non-nilpotent. Using GAP, it is easy to see that \G\ = 8!, so G = Ss- □ 
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We now discuss the situation of Example 14.31 Clearly, if the pair (x, y) G 
G X G satisfies the condition 

[X, 42/] = [y, 4X] = 1, (*) 

then all conjugates {x^,y^), for all g £ G, satisfy the analogous property. 
Therefore it is sensible to consider classes under conjugation. 

It turns out by GAP that the only pairs {x,y) £ GxG satisfying (*), that 
generate a non-nilpotent subgroup of G, have both x and y with cycle struc- 
ture of type (4) (2) (2) and, in addition, x,y necessarily generate the whole 
group G. Without loss of generality, we may assume x = (1, 2, 3, 4) (5, 6) 
(7, 8). For this x, we calculated all solutions y G G of (*). We ended up with 
precisely 64 solutions. Of course, the group Cg{x) of order 32 acts on the 
pairs of solutions. The stabilizer of this action is Cg{x) fl Cciy) = Z{G) = 1, 
so that we obtain two essentially distinct solutions. 

Other examples? Suppose that in some finite group we can find Sylow 
p-subgroups P, Q and elements x £ P,y £ Q such that [x,y] £ P (1 Q. Let 
c be the nilpotency class of P. Thus, [x, c+iv] = [y, c+ix] = 1. If xy is not 
a p-element, then {x,y) is non-nilpotent. The groups in Examples 14.21 and 
14.31 are of this form for p = 2. It would be very interesting to find analogous 
examples for all odd primes p. 

References 

[1] A. Abdollahi, Engel graph associated with a group, J. Algebra 318 
(2007), 680-691. 

[2] D. J. Collins and A. Juhasz, Engel groups III, Israel J. Math. 174 
(2009), 73-91. 

[3] G. Endimioni and G. Traustason, Groups that are pairwise nilpotent. 
Comm. Algebra 36 (2008), 4413-4435. 

[4] The GAP Group, GAP - Groups, Algorithms, and Programming, Version 
4.4.12, 2008; |http : //www . gap- system . org| 

[5] E. S. Golod, Some problems of Burnside type, 1968 Proc. Int. Congr. 
Math. (Moscow 1966), 284-289; Amer. Math. Soc. Transl. (2) 84 
(1969), 83-88. 

[6] W. Nickel, Computation of nilpotent Engel groups, J. Austral. Math. 
Soc. Ser. A 67 (1999), 214-222. 

[7] D. J. S. Robinson, Finiteness conditions and generalized soluble groups. 
Part 1 and 2, Springer-Verlag, Berlin, 1972. 



8 



